Signal processing apparatus, imaging apparatus, signal processing method and program for correcting deviation of blurring in images

ABSTRACT

Provided is a signal processing apparatus, including an input unit into which first image data and second image data are input, the first image data and the second image data being obtained by capturing a predetermined subject with an imaging unit and having mutually different states of blurring; a distance information calculation unit that calculates first distance information in accordance with a position in a depth direction of the subject based on the first image data and the second image data; and a distance information correction unit that calculates second distance information by performing, on the first distance information, processing to correct a deviation of the blurring caused by a mismatch between an image-forming plane of the subject and an imaging plane of the imaging unit.

BACKGROUND

The present disclosure relates to, for example, a signal processingapparatus that appropriately calculates distance information, an imagingapparatus, a signal processing method, and a program.

In recent years, a small-area image sensor is frequently used forcompact digital still cameras due to a demand for compactness. Forexample, a 1/2.33 type CCD (6.2×4.6) is used. As large-format imagesensors, on the other hand, the 35-mm film size (36×24) and the APS-Csize (23.4×16.7) are known.

In a compact digital still camera with such a small screen size, thedepth of field becomes deep and a blurring effect normally obtained by aconventional film camera and a digital single-lens reflex camera using alarge-format image sensor is extremely small. The blurring effect isdemanded for portrait imaging in which a person is made to stand out byblurring the background. Against the above background, as described inJapanese Patent Application Laid-Open No. 2007-66199, techniques toobtain a large blurring effect even in a digital still camera using asmall image sensor have been proposed.

According to the method described in Japanese Patent ApplicationLaid-Open No. 2007-66199, faces and persons are detected from a portraitimage shot and locations other than faces and persons are considered tobe the background and blurring processing is performed thereon.

SUMMARY

The method described in Japanese Patent Application Laid-Open No.2007-66199 applies blurring processing uniformly to the background.However, different amounts of blurring should be added to subjectscontained in the background in different distances and the methoddescribed in Japanese Patent Application Laid-Open No. 2007-66199 posesa problem from the viewpoint of adding natural blurring. Further, themethod described in Japanese Patent Application Laid-Open No. 2007-66199poses a problem that it is difficult to apply the method to othersubjects than persons. Further, the method described in Japanese PatentApplication Laid-Open No. 2007-66199 poses a problem that a deviation ofblurring caused by a mismatch between an image-forming plane of asubject and an imaging plane of an imaging unit such as a curvature offield is not taken into consideration.

Therefore, it is desirable to provide a signal processing apparatus thatcorrects an error of distance information caused by, for example, adeviation of blurring, an imaging apparatus, a signal processing method,and a program.

To solve the above problems, the present disclosure is, for example, asignal processing apparatus including an input unit into which firstimage data and second image data are input, the first image data and thesecond image data being obtained by capturing a predetermined subjectwith an imaging unit and having mutually different states of blurring, adistance information calculation unit that calculates first distanceinformation in accordance with a position in a depth direction of thesubject based on the first image data and the second image data, and adistance information correction unit that calculates second distanceinformation by performing, on the first distance information, processingto correct a deviation of the blurring caused by a mismatch between animage-forming plane of the subject and an imaging plane of the imagingunit.

The present disclosure is, for example, a imaging apparatus, includingan imaging unit, an input unit into which first image data and secondimage data are input, the first image data and the second image databeing obtained by capturing a predetermined subject with the imagingunit and having mutually different states of blurring, a distanceinformation calculation unit that calculates first distance informationin accordance with a position in a depth direction of the subject basedon the first image data and the second image data, and a distanceinformation correction unit that calculates second distance informationby performing, on the first distance information, processing to correcta deviation of the blurring caused by a mismatch between animage-forming plane of the subject and an imaging plane of the imagingunit.

The present disclosure is, for example, a signal processing method,including inputting first image data and second image data, the firstimage data and the second image data being obtained by capturing apredetermined subject with an imaging unit and having mutually differentstates of blurring, calculating first distance information in accordancewith a position in a depth direction of the subject based on the firstimage data and the second image data, and calculating second distanceinformation by performing, on the first distance information, processingto correct a deviation of the blurring caused by a mismatch between animage-forming plane of the subject and an imaging plane of the imagingunit.

The present disclosure is, for example, a program causing a computer toexecute a signal processing method, including inputting first image dataand second image data, the first image data and the second image databeing obtained by capturing a predetermined subject with an imaging unitand having mutually different states of blurring, calculating firstdistance information in accordance with a position in a depth directionof the subject based on the first image data and the second image data,and calculating second distance information by performing, on the firstdistance information, processing to correct a deviation of the blurringcaused by a mismatch between an image-forming plane of the subject andan imaging plane of the imaging unit.

According to at least one embodiment, high-precision distanceinformation having a minor error can be obtained by performingprocessing that takes a deviation of blurring into consideration.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram exemplifying the configuration of a systemaccording to a first embodiment of the present disclosure;

FIG. 2 is a block diagram exemplifying a main configuration of a signalprocessing apparatus according to the first embodiment of the presentdisclosure;

FIG. 3 is a block diagram exemplifying a detailed configuration of acontrol unit of the signal processing apparatus according to the firstembodiment of the present disclosure;

FIG. 4 is a flow chart illustrating an example of a processing overviewaccording to an embodiment of the present disclosure;

FIG. 5 is a flow chart illustrating an example of processing tocalculate first distance information according to an embodiment of thepresent disclosure;

FIG. 6 is a schematic diagram illustrating an example of the processingto calculate the first distance information according to an embodimentof the present disclosure;

FIG. 7 is a schematic diagram used for describing a Pill Box function;

FIG. 8 is a schematic diagram used for describing the Gaussian filter;

FIG. 9 is a schematic diagram illustrating an example of the processingto calculate the first distance information according to an embodimentof the present disclosure;

FIG. 10 is a schematic diagram illustrating an example of a curvature offield;

FIG. 11 is a schematic diagram illustrating an example of a blur circlecaused by the curvature of field;

FIG. 12 is a schematic diagram illustrating another example of the blurcircle caused by the curvature of field;

FIG. 13 is a schematic diagram illustrating an example of single-sidedblurring;

FIG. 14 is a schematic diagram illustrating another example of thesingle-sided blurring;

FIG. 15 is a schematic diagram showing an example of the curvature offield;

FIG. 16 is a schematic diagram illustrating an error of a distance mapcaused by an example of the curvature of field;

FIG. 17 is a schematic diagram showing another example of the curvatureof field;

FIG. 18 is a schematic diagram illustrating an error of the distance mapcaused by another example of the curvature of field;

FIG. 19 is a schematic diagram showing an example of the single-sidedblurring;

FIG. 20 is a schematic diagram illustrating an error of the distance mapcaused by an example of the single-sided blurring;

FIG. 21 is a flow chart illustrating processing to correct distanceinformation according to the first embodiment of the present disclosure;

FIG. 22 is a schematic diagram illustrating the definition of a Defocusamount according to the first embodiment of the present disclosure;

FIG. 23 is a flow chart illustrating processing to correct the distanceinformation according to a second embodiment of the present disclosure;

FIGS. 24A and 24B are schematic diagrams illustrating foregroundreplacement processing according to the second embodiment of the presentdisclosure;

FIG. 25 is a schematic diagram showing a plurality of examples of thecurvature of field;

FIG. 26 is a schematic diagram exemplifying a base by a first maincomponent of the curvature of field;

FIG. 27 is a schematic diagram exemplifying the base by a second maincomponent of the curvature of field;

FIG. 28 is a schematic diagram exemplifying the base of the single-sidedblurring in a vertical direction;

FIG. 29 is a schematic diagram exemplifying the base of the single-sidedblurring in a horizontal direction;

FIG. 30 is a block diagram exemplifying the configuration of an imagingapparatus according to a third embodiment of the present disclosure; and

FIG. 31 is a block diagram showing a portion of the configuration of theimaging apparatus according to the third embodiment of the presentdisclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

The embodiments of the present disclosure will be described below. Thedescription will be provided in the order shown below:

<1. First Embodiment> <2. Second Embodiment> <3. Third Embodiment> <4.Modification>

However, the present disclosure is not limited to the embodiments andmodification described below.

1. First Embodiment Overall Configuration of System

The first embodiment of the present disclosure will be described. FIG. 1shows an example of a system according to the first embodiment of thepresent disclosure. A system 1 is constituted of, for example, a signalprocessing apparatus 2 and an external device 3 connected to the signalprocessing apparatus 2. The signal processing apparatus 2 is a devicesuch as a personal computer and TV set. The external device 3 is, forexample, an imaging apparatus having an imaging function such as adigital still camera. Bidirectional communication by wire or by radio isperformed between the signal processing apparatus 2 and the imagingapparatus 3. For example, a plurality of pieces of image data containingtwo pieces of image data in different blurred states and specificinformation such as an f-number of the imaging apparatus 3 aretransmitted from the imaging apparatus 3 to the signal processingapparatus 2 by the communication. The image data is, for example, dataobtained by a specific subject being imaged by an imaging unit of theimaging apparatus 3.

“Configuration of the Signal Processing Apparatus”

FIG. 2 shows an example of a main configuration of the signal processingapparatus 2. The signal processing apparatus 2 includes, for example, aninterface (I/F) 4 between the signal processing apparatus 2 and theimaging apparatus 3, a control unit 5, and a blurring processingapplication unit 6. The signal processing apparatus 2 may include otherconfigurations. For example, the signal processing apparatus 2 may beconfigured to include an operation unit such as a keyboard and a mouse,a display unit, or a speaker. Further, the signal processing apparatus 2may also be configured to include a communication unit to connect to anetwork such as the Internet. A plurality of pieces of image datacontaining two pieces of image data in different blurred states may besupplied to the signal processing apparatus 2 from an external deviceconnected to a network via the communication unit.

The control unit 5 is configured by, for example, a CPU (CentralProcessing Unit). A ROM (Read Only Memory) or a RAM (Random AccessMemory) may be connected to the control unit 5. The control unit 5performs predetermined processing according to, for example, a programstored in the ROM. The control unit 5 performs, for example, processingto generate a distance map by determining distance information andprocessing to correct the distance map.

The blurring processing application unit 6 performs blurring processingon image data by using a distance map supplied by the control unit 5.The function of the blurring processing application unit 6 may beincorporated into the control unit 5. Further, the processing by theblurring processing application unit 6 may be performed by anotherapparatus separate from the signal processing apparatus 2.

“Configuration of the Control Unit”

FIG. 3 shows an example of a detailed configuration of the control unit5.

The control unit 5 is constituted of, for example, an image input unit 5a, a distance information calculation unit 5 b, and a distanceinformation correction unit 5 c. For example, a plurality of pieces ofimage data containing two pieces of image data in different blurredstates is input into the image input unit 5 a. The plurality of piecesof image data is supplied to the distance information calculation unit 5b from the image input unit 5 a. The plurality of pieces of suppliedimage data is stored in the RAM or the like.

One piece of image data of the plurality of pieces of image data isimage data (first image) focused on a subject, for example, a person inthe foreground. Another piece of image data of the plurality of piecesof image data is image data (second image) of the same subjectphotographed by shifting the focus to the far side by a predetermineddistance with respect to the first image. Still another piece of imagedata of the plurality of pieces of image data is image data (thirdimage) of the same subject photographed by shifting the focus to thenear side by the predetermined distance with respect to the first image.

The three images are images having mutually different degrees ofblurring. The foreground is in focus in the first image and thus, thedegree of the foreground being out of focus in the second image and thedegree of the foreground being out of focus in the third image areequal. The degree of blurring of the background in the third image islarger than the degree of blurring of the background in the secondimage. Image data corresponding to the three images is supplied to, forexample, the distance information calculation unit 5 b. Image datacorresponding to the first image may directly be supplied to theblurring processing application unit 6 without going through the controlunit 5.

The distance information calculation unit 5 b performs processing todetermine the first distance information by using supplied image data.The distance information is a distance in accordance with the positionof a subject in a depth direction and, for example, the distance betweenthe foreground and the background. The distance information isdetermined in units of, for example, pixels of image data. The distanceinformation may also be determined in units of blocks of image data. Thedistance information is determined for each pixel by the distanceinformation calculation unit 5 b to generate a distance map beforecorrection constituted of distance information for all pixels.

The distance map generated by the distance information calculation unit5 b is supplied to the distance information correction unit 5 c. Thedistance information correction unit 5 c performs processing to correctdistance information for each pixel of the supplied distance map. Thedistance information correction unit 5 c performs correction processingto correct the deviation of blurring on, for example, distanceinformation for each pixel. Distance information after the correction,which is second distance information, is determined by the correctionprocessing. Distance information of all pixels is corrected to generatea distance map after the correction (corrected distance map) constitutedof corrected distance information. The generated corrected distance mapand image data corresponding to the first image data are supplied to theblurring processing application unit 6. The blurring processingapplication unit 6 refers to the supplied corrected distance map toperform blurring processing on the image data corresponding to the firstimage data.

“Overall Flow of the Processing”

The overall flow of the processing will be described by referring toFIG. 4. In FIG. 4, for example, processing of steps S1 to S5 isperformed by the imaging apparatus 3 and processing of steps S6 to S8 isperformed by the signal processing apparatus 2.

In the first step S1, the first image is captured. In the case of, forexample, portrait, the image is captured in such a way that a person inthe foreground is in focus. The first image is temporarily stored. Next,in step S2, the position of focus is changed by a predetermined amountto the far side with respect to the first image. Then, in step S3, thesecond image is captured. The second image is temporarily stored.

Next, in step S4, the position of focus is changed by the predeterminedamount to the near side with respect to the first image. The change ofthe position of focus is assumed to be equal to the change when thesecond image is acquired. Then, in step S5, the third image is captured.The captured three images (the first image, second image, and thirdimage) are supplied to the distance information calculation unit 5 b viathe image input unit 5 a.

Next, in step S6, for example, distance information (first distanceinformation) for each pixel is calculated by using the second image andthird image. The processing in step S6 is performed by, for example, thedistance information calculation unit 5 b. The, a distance mapconstituted of distance information for all pixels is generated. Thegenerated distance map is supplied to the distance informationcorrection unit 5 c. In the description that follows, calculatingdistance information may be denoted as distance estimation or estimatingthe distance.

Next, in step S7, processing to correct the distance map may beperformed. The processing in step S7 is performed by, for example, thedistance information correction unit 5 c. In step S7, the distanceinformation for each pixel in the supplied distance map is corrected togenerate a corrected distance map constituted of distance information ofall pixels after the correction. The generated corrected distance map issupplied to the blurring processing application unit 6. Lastly, in stepS8, blurring processing is applied to the first image while thecorrected distance map being referred to. The processing in step S8 isperformed by, for example, the blurring processing application unit 6.

For the processing in step S8, for example, a low-pass filter can beused. Whether a low-pass filter is applicable and the strength thereof(the lower the cutoff frequency, the stronger the strength thereof) areset in accordance with distance information of each pixel in thecorrected distance map. That is, for example, the low-pass filter is notapplied to the foreground (person) and the low-pass filter of thestrength in accordance with the distance from the foreground is appliedto the background. A portrait image with a blurred background suitablefor appreciation can be generated by the blurring processing.Incidentally, the foreground is not limited to persons and is setappropriately in accordance with the subject. For example, an animal orfruit may be the foreground.

[Distance Map Generation Processing]

Step S6 (distance map generation) in FIG. 4 will be described in moredetail with reference to FIGS. 5 and 6. FIG. 6 shows the second imageand the third image. First, as shown in FIG. 6, any pixel in the secondimage is selected and coordinates thereof are set as (x0, y0). The rangeof (p×p) is defined by p, which is a sufficiently large value comparedwith the value of q described later, by setting (x0, y0) as the centerthereof. A convolution operation by the filter is performed for pixelvalues contained in the range of (p×p) of the second image (step S11 inFIG. 5). For example, a convolution operation using a predeterminedGaussian filter is performed. The Gaussian filter is a blurring filterusing a Gaussian function. That is, in the Gaussian filter, the weightfor combining pixels is decided by a Gaussian function in accordancewith the distance from the center pixel.

Next, in step S12, a similarity between the pixel value in the range of(q×q) around (x0, y0) of the resultant pixel value in step S11 and thepixel value in the range of (q×q) around (x0, y0) of the third image iscalculated. As the similarity, SAD (Sum of Absolute Difference) or thelike can be used. For example, SAD is a value obtained by determining anabsolute value of a difference of two pixel values corresponding to thesame position between the range of (q×q) of the second image and therange of (q×q) of the third image and integrating the absolute value ofthe difference for (q×q) pixels. The value of SAD decreases with anincreasing similarity.

In determination processing (step S13) after step S12, whether thecalculation of similarity and processing to store the value ofsimilarity have been performed M times is determined. M is asufficiently large value. If it is determined that the calculation ofsimilarity and processing to store the value of similarity have beenperformed M times, the processing proceeds to step S14. In step S14, themaximum value (the minimum value when SAD is used as the similarity) ofsimilarity among stored similarities for M times is searched for. Anumber of repetitions k in step S11 and step S12 corresponding to themaximum number of similarity is set as distance information (firstdistance information). In the number of times of blurring processing bythe Gaussian filter, the number of times of processing in which blurringis not added may be set to 0. When a similarity is calculated, acomparison is also made before blurring processing by the Gaussianfilter is performed.

Areas in the foreground of the second image and the third image haveapproximately similar blurring added thereto and thus, SAD is determinedto be small and the similarity is determined to be large. The degree ofblurring of the background in the third image is larger than the degreeof blurring of the background in the second image and thus, thesimilarity increases by performing processing to add blurring on thesecond image.

As a method of determining the maximum value of similarity, a method ofnot repeating processing of the preset M times is also possible. If, forexample, SAD is used, a minimum value of SAD may be detected to set thenumber of times k before the minimum value of SAD is generated asdistance information. Further, instead of the absolute value ofdifference, the square of difference may be used as the similarity.Further, the similarity may be detected by using other parameters thanSAD.

As described above, k is determined for each of all pixels in an image.In a distance map, the determined value of k is present for each pixel.In this case, a distance map in which the value of k is determined forpixels sampled at predetermined intervals in the horizontal directionand/or the vertical direction, instead of all pixels, may be created toreduce the amount of computation.

To reduce the amount of computation to decrease the load of processing,distance information may be determined for each predetermined range. Thepredetermined range can appropriately be set like 3 pixels×3 pixels and5 pixels×5 pixels. A representative pixel in the predetermined range isselected. The pixel positioned in the center of the predetermined rangeor the like can appropriately be set as the representative pixel. Then,the distance information k of the representative pixel is calculated bythe above method. The calculated distance information k about therepresentative pixel may be used as distance information for each pixelcontained in the predetermined range.

If the range is, for example, 3 pixels×3 pixels, 5 pixels×5 pixels orso, the distance information k of adjacent pixels does not changesignificantly. Therefore, instead of individually determining thedistance information k for each pixel in the predetermined range, almostno practical problem is caused by approximation using the distanceinformation k of the representative pixel. As another method, thedistance information k may be determined by averaging a parameter todetermine the distance information k in the predetermined range andusing the average of the parameter.

Next, why the value of k can be distance information will be described.First, the following symbols are defined:

Img: Image in focus to which a blurring function by defocusing is notapplied

σ₂, σ₃: Standard deviation of the blurring function (two-dimensionalisotropic Gaussian function) in the second image and the third imagerespectively

σ_(f): Standard deviation of the Gaussian filter applied in step S11

The blurring function by defocusing is geometrical-optically determinedas a Pill Box function having a blur circle diameter as shown in FIG. 7.When a point image is in focus, the point image is generated on theimaging plane, but when the point image is out of focus, a circularimage is generated by blurring. This circle is called a blur circle.

For an actual lens, however, in consideration of the influence ofaberration and the like, the blurring function is frequentlyapproximated as a two-dimensional isotropic Gaussian function as shownin FIG. 8. In FIG. 8, the center position is the position of a pixel(attention pixel) to be processed. The Gaussian filter has adistribution (function of the Gaussian distribution) in which the weightwhen an average value is calculated increases with a decreasing distanceto the position of the attention pixel and the weight decreases with anincreasing distance from the pixel in the attention position. Thesmaller the standard deviation and the narrower the width ofdistribution, the smaller the effect of smoothing.

Also in the first embodiment of the present disclosure, the blurringfunction when the second image and the third image are defocused isapproximated as a two-dimensional isotropic Gaussian function and thestandard deviation thereof is assumed to be σ₂ and σ₃ respectively. Inthis case, the process to determine k in the processing shown in FIG. 5corresponds to the determination of k by the following formula (1) (G(μ,σ) is a two-dimensional isotropic Gaussian function of the average μ andthe standard deviation σ). Further, the variance is a squared expectedvalue of shifts from the average.

$\begin{matrix}{{{Im}\mspace{11mu} {g \otimes {G\left( {0,\sigma_{3}} \right)}}} = {{Im}\mspace{11mu} {g \otimes {G\left( {0,\sigma_{2}} \right)} \otimes \underset{\underset{k}{}}{G{\left( {0,\sigma_{f}} \right) \otimes {G\left( {0,\sigma_{f}} \right)}}\mspace{14mu} {\ldots \; \otimes {G\left( {0,\sigma_{f}} \right)}}}}}} & (1)\end{matrix}$

By using the fact that a convolution of Gaussian functions generallybecomes a Gaussian function and the sum of variances of two Gaussianfunctions to be convoluted (σ₂ ²+σ₃ ²) matches the variance of theresultant Gaussian function of the convolution, k can be described bythe following formula (2):

$\begin{matrix}{k = {\frac{\sigma_{3}^{2} - \sigma_{2}^{2}}{\sigma_{f}^{2}} = \frac{\left( {\sigma_{3} - \sigma_{2}} \right)\left( {\sigma_{3} + \sigma_{2}} \right)}{\sigma_{f}^{2}}}} & (2)\end{matrix}$

From the formula (2), k has the following relationships. (1) k isinversely proportional to the square of σ_(t). If σ_(f) is a constantvalue, the denominator of the formula (2) becomes a constant value.

(2) k is proportional to (σ₃−σ₂). (σ₃−σ₂) can approximately beinterpreted as a difference of blur circle diameters of the blurringfunction of defocusing in the second image and the third image. Thisvalue becomes a constant value if the amounts of defocusing of thesecond image and the third image with respect to the first image aremaintained constant. That is, if settings are made so that, for example,the second image is defocused by 2DoF (depth of field) to the far sidewith respect to the first image and the third image is defocused by 2DoFto the near side with respect to the first image, (σ₃−σ₂) becomes aconstant value of four times the value of the permissible blur circle.

(3) k is proportional to (σ₃+σ₂). (σ₃+σ₂) can approximately beinterpreted as a sum or twice the average value of blur circle diametersof the blurring function of defocusing in the second image and the thirdimage.

For example, as described above, the second image is defocused by 2DoFto the far side with respect to the first image and the third image isdefocused by 2DoF to the near side with respect to the first image. Ifthe relative distance between the foreground position to be focused whenthe first image is acquired and the background position is nDoF, σ₂becomes the value (n−2) times the permissible blur circle and σ₃ becomesthe value (n+2) times the permissible blur circle and thus, (σ₃+σ₂)becomes the value 2n times the permissible blur circle and therefore,(σ₃+σ₂) is proportional to the relative distance n between theforeground position and the background position.

From the above relationships, k has a property of being proportional tothe relative distance n between the foreground position and thebackground position in the end. Due to this property, distanceinformation proportional to the relative distance between the foregroundand the background can be obtained by calculating k. FIG. 9 showsrelationships among k, n, and (σ₃+σ₂). That is, FIG. 9 schematicallyshows the relationship k∝Σ₃+σ₂∝n (∝ indicates a linear relationship).

“Blurring Processing”

The calculated distance information k is corrected by the processing instep S7 of FIG. 4. The processing in step S7 will be described later. Instep S8 of FIG. 4, blurring processing is performed by the blurringprocessing application unit 6 in accordance with the calculated value ofk after the correction in each pixel position in the image. That is, ifthe value of k after the correction is large, stronger blurringprocessing is performed and if the value of k after the correction issmall, weaker blurring processing or no blurring processing isperformed. By performing such blurring processing, an image suitable forappreciation with an appropriately blurred background can be obtained.

By using the calculated k after the correction, as will be describedbelow, the blurring processing in step S8 may be performed bydetermining the relative physical distance s_(f) between the foregroundposition and the background position and using s_(f). First, thefollowing formula (3) is generally derived from a formula of geometricaloptics of a thin lens.

$\begin{matrix}{s_{f} = {\frac{f^{2}\left( {s - f} \right)}{f^{2} - {\delta \; {F\left( {s - f} \right)}}} + f}} & (3)\end{matrix}$

In the formula (3), f is a focal length, F is an f-number, s is afocusing distance, δ is a blur circle diameter, and s_(f) is a far-sidesubject distance. f, F, and s are control values of a lens and known andif δ is known, s_(f) can be determined. Geometrical-optically, ablurring function has some blur circle diameter represented as a PillBox function and here, the average value of σ₃ and σ₂ determined fromthe formula (2) is considered to be approximately the blur circlediameter. The formula (4) is substituted as δ in the formula (3).

$\begin{matrix}{\frac{\sigma_{3} + \sigma_{2}}{2} = \frac{k\; \sigma_{f}^{2}}{2\left( {\sigma_{3} - \sigma_{2}} \right)}} & (4) \\{s_{f} = {\frac{f^{2}\left( {s - f} \right)}{f^{2} - \frac{k\; \sigma_{f}^{2}{F\left( {s - f} \right)}}{2\left( {\sigma_{3} - \sigma_{2}} \right)}} + f}} & (5)\end{matrix}$

The blurring processing in step S8 may be performed by using a relativeseparation distance Sf between the foreground position and thebackground position determined by the formula (5).

“Influence by the Deviation of the Blurring Function”

k or s_(f) derived heretofore contains, as represented by the formula(1), a precondition that the blurring function is a two-dimensionalisotropic Gaussian function. Though, as described above, the blurringfunction is geometrical-optically determined as a Pill Box function, butin consideration of the influence of diffraction and aberration of anactual lens, the blurring function is frequently approximated as atwo-dimensional isotropic Gaussian function.

Further, as a precondition, the blurring function is assumed to beconstant. However, a deviation is caused by a mismatch of theimage-forming plane of a subject and the imaging plane in the blurringfunction of an actual lens. In other words, the blurring function of anactual lens is different in magnitude (the blur circle diameter or ifthe blurring function is considered as a Gaussian function, the standarddeviation thereof) in accordance with the position on the imaging plane.An increasing error arises in the distance estimation with an increasingdeviation of the blurring function (with an increasing difference inmagnitude of the blurring function). Various kinds of aberration andvariations in production and assembly of lenses can be cited as causesof a deviation of the blurring function. Among these causes,particularly “curvature of field” and “single-sided blurring” can becited as causes of an increasing deviation of the blurring function.“Curvature of field” and “single-sided blurring” will be describedbelow.

“Curvature of Field”

The curvature of field is a typical aberration. FIG. 10 schematicallyshows how a curvature of field is generated. The imaging planecorresponds to the position and/or shape of an image sensor like, forexample, CCD (Charge Coupled Device) and CMOS (complementary Metal OxideSemiconductor). The imaging plane is, for example, a plane linkingphotoelectric conversion positions where photoelectric conversion isperformed in an image sensor. The image-forming plane is, for example, aplane linking image-forming positions of each point on a planar subjectplane. If a subject in any position on a planar subject plane is formedas an image in focus on a plane imaging plane, no curvature of field isgenerated. However, an actual lens has a curved image-forming plane andthe image-forming plane and the imaging plane do not match, causing amismatch. This kind of aberration is called a curvature of field.

If a curvature of field arises, the captured image is in focus in, forexample, the center of the imaging plane, but blurring arises inportions other than the center in accordance with the amount ofcurvature of field. An example of this state is shown in FIG. 11.Further, if the center of the imaging plane is out of focus, theblurring function is different in magnitude between the center of theimaging plane and other portions than the center. An example of thisstate is shown in FIG. 12. The amount of curvature of field and aresultant amount of blurring of a captured image are rotationallysymmetric with respect to an optical axis from the generation principlethereof.

“Single-Sided Blurring”

Single-sided blurring is caused by, for example, variations inproduction and assembly of lenses, variations in assembly of lenses andimage sensors, or combined factors thereof. FIGS. 13 and 14schematically show how single-sided blurring is generated. The exampleof single-sided blurring shown in FIG. 13 is an example in whichsingle-sided blurring is caused by variations in production and assemblyof lenses. Due to variations in production and assembly of lenses, theimage-forming plane linking image-forming positions of each point on aplanar subject plane perpendicular to the optical axis does not matchthe imaging plane perpendicular to the optical axis. Though similar tothe curvature of field, the amount of shift of the image-formingposition from the imaging plane is not rotationally symmetric and thisis different from the case caused by aberration.

FIG. 14 shows an example of single-sided blurring generated byvariations in assembly of lenses and image sensors. Though theimage-forming plane is perpendicular to the optical axis, theimage-forming plane and the imaging plane do not match due to thepresence of the image sensor. In both cases of FIGS. 13 and 14, thecaptured image is in focus near the center of the imaging plane, butblurring arises in portions other than the center in accordance with theamount of single-sided blurring. Further, if the center of the imagingplane is not in focus, the magnitude of the blurring function isdifferent depending on the position on the imaging plane.

“Error of Distance Information”

If the curvature of field or single-sided blurring described abovearises, an error arises in the calculated distance information. Thereason therefor will be described. First, the distance estimationwithout error can be done in the center of the image plane by the aboveformula (2) and the distance information k without error can beobtained. In the periphery of the image plane, on the other hand, adeviation Δσ arises in σ₂ and σ₃ due to a curvature of the image-formingplane. For example, the standard deviations of the blurring function inthe periphery of the image plane are assumed to be given by formulas (6)and (7).

σ₂′=σ₂+Δσ  (6)

σ₃′=σ₃+Δσ  (7)

If the formulas (6) and (7) are substituted into the formula (2) andrearranged, obtained distance information k′ is given by the followingformula (8):

$\begin{matrix}\begin{matrix}{k^{\prime} = \frac{\left( {\sigma_{3}^{\prime} - \sigma_{2}^{\prime}} \right)\left( {\sigma_{3}^{\prime} + \sigma_{2}^{\prime}} \right)}{\sigma_{f}^{2}}} \\{= \frac{\begin{matrix}\left\{ {\left( {\sigma_{3} + {\Delta \; \sigma}} \right) - \left( {\sigma_{2} + {\Delta \; \sigma}} \right)} \right\} \\\left\{ {\left( {\sigma_{3} + {\Delta \; \sigma}} \right) + \left( {\sigma_{2} + {\Delta \; \sigma}} \right)} \right\}\end{matrix}}{\sigma_{f}^{2}}} \\{= \frac{\left( {\sigma_{3} - \sigma_{2}} \right)\left( {\sigma_{3} + \sigma_{2} + {2\Delta \; \sigma}} \right)}{\sigma_{f}^{2}}}\end{matrix} & (8)\end{matrix}$

From the formula (8), a positive-side error 2Δσ arises for the distanceinformation k in the center of the image plane.

Also, for example, the standard deviations of the blurring function inthe periphery of the image plane are assumed to be given by formulas (9)and (10).

σ₂′=σ₂−Δσ  (9)

σ₃′=σ₃−Δσ  (10)

If the formulas (9) and (10) are substituted into the formula (2) andrearranged, obtained distance information k′ is given by the followingformula (11):

$\begin{matrix}\begin{matrix}{k^{\prime} = \frac{\left( {\sigma_{3}^{\prime} - \sigma_{2}^{\prime}} \right)\left( {\sigma_{3}^{\prime} + \sigma_{2}^{\prime}} \right)}{\sigma_{f}^{2}}} \\{= \frac{\begin{matrix}\left\{ {\left( {\sigma_{3} - {\Delta \; \sigma}} \right) - \left( {\sigma_{2} - {\Delta \; \sigma}} \right)} \right\} \\\left\{ {\left( {\sigma_{3} - {\Delta \; \sigma}} \right) + \left( {\sigma_{2} - {\Delta \; \sigma}} \right)} \right\}\end{matrix}}{\sigma_{f}^{2}}} \\{= \frac{\left( {\sigma_{3} - \sigma_{2}} \right)\left( {\sigma_{3} + \sigma_{2} - {2\Delta \; \sigma}} \right)}{\sigma_{f}^{2}}}\end{matrix} & (11)\end{matrix}$

From the formula (11), a negative-side error 2Δσ arises for the distanceinformation k in the center of the image plane.

FIG. 15 shows an example of the curvature of field. FIG. 16 shows anexample of a distance map obtained when the curvature of field shown inFIG. 15 arises. As an example of the subject, a subject in which a planebackground in the same distance is arranged in the far distance and arectangular object (foreground) is arranged in the near distance isassumed. In shades of color of a distance map, the denser (higher) theconcentration, the nearer an object and the thinner (lower) theconcentration, the farther an object.

As shown in FIG. 16, an object in close range positioned in the centerof the image plane has a high concentration on a distance map, providingappropriate distance information. Further, the concentration on thedistance map near the center of the image plane is low, providingappropriate distance information. In the periphery (edge) of the imageplane, on the other hand, due to the influence of error described above,the far side, that is, a low concentration should be realized, but thenear side, that is, a high concentration is realized. If the blurringprocessing shown in step S8 of FIG. 4 is performed by referring to sucha distance map, the amount of blurring in the background portion whereblurring processing should be applied will be lower than the appropriateamount. As a result, an image inappropriate for appreciation isgenerated.

The curvature of field is rotationally symmetric with respect to theoptical axis, but the amount of curvature of field in each image planeposition is different depending on the lens design and lens state (suchas the focal length, f-number, and focus position). FIG. 17 is anotherexample of the curvature of field. FIG. 18 shows an example of thedistance map obtained when the curvature of field shown in FIG. 17arises.

In FIG. 18, appropriate distance information is obtained in the centerof the image plane and the neighborhood thereof and the distanceestimation is done accurately. However, due to the influence of errordescribed above, shades of concentration arise concentrically. If such adistance map is referred to, it is difficult to perform appropriateblurring processing and, as a result, an image inappropriate forappreciation is generated.

FIG. 19 shows an example of single-sided blurring. FIG. 20 shows anexample of the distance map obtained when the single-sided blurringshown in FIG. 19 arises. In the distance map shown in FIG. 20, thedistance estimation value to the background is not constant, causing anerror in the up and down direction of the image plane. While thesingle-sided blurring in the up and down direction is illustrated here,but the single-sided blurring can arise also in a left and rightdirection or an oblique direction depending on variations of lenses.

“Distance Map Correction Processing in the First Embodiment”

To eliminate the influence of an error of distance information caused bya deviation of the blurring function described above, distance mapcorrection processing described below is performed in the firstembodiment of the present disclosure. The processing is performed by,for example, the distance information correction unit 5 c andcorresponds to the processing in step S7 of FIG. 4.

FIG. 21 is a flow chart exemplifying the distance map correctionprocessing. The distance information k′, which is the first distanceinformation, is obtained for each pixel by the processing in step S6 ofFIG. 4. Then, a distance map k′(x, y) constituted of the distanceinformation k′ for all pixels is obtained. (x, y) indicates coordinateson the image plane.

The distance map correction processing in the first embodimentcalculates the correction value involved in a deviation of blurring byusing Defocus information B(x, y), which is known information. Then,correction processing (operation) using the correction value isperformed on the distance information k′ to calculate distanceinformation k_(—comp), which is second distance information. Thedistance information k_(—comp) is calculated for all pixels to generatea corrected distance map k_(—comp)(x, y) constituted of the distanceinformation k_(—comp) for all pixels.

FIG. 22 shows Defocus information B(x, y), which is an example of knowninformation. The Defocus information B(x, y) is a physical distancerepresenting a deviation amount of the imaging plane and theimage-forming plane at each coordinate point on the image plane. TheDefocus amount caused by a curvature of field is fixed when a lens isdesigned and is known. The Defocus amount caused by single-sidedblurring is obtained as known information by, for example, photographingby using a resolution chart when a lens is manufactured and measuringthe amount of blurring of the chart in each position of the image plane.

The Defocus information B(x, y) is stored in, for example, a storageunit such as a ROM connected to the control unit 5. The Defocusinformation B(x, y) may be transmitted from the imaging apparatus 3 tothe signal processing apparatus 2 as a piece of specific information.The Defocus information B(x, y) may also be supplied to the signalprocessing apparatus 2 via a network.

To set the value of k_(—comp)(x, y) to k regardless of the position, asshown in the formula (12), a difference between the formulas (2) and (8)may be corrected. If the formulas (8) and (2) are substituted into k′(x,y) and k(x, y) respectively and rearranged, the following formula (12)is obtained. Because A is different depending on the position on theimage plane, Δσ is shown as Δσ(x, y) in the form of a function.

$\begin{matrix}\begin{matrix}{{k_{comp}\left( {x,y} \right)} = {{k^{\prime}\left( {x,y} \right)} - \left\{ {{k^{\prime}\left( {x,y} \right)} - {k\left( {x,y} \right)}} \right\}}} \\{= {{k^{\prime}\left( {x,y} \right)} - {\frac{2\left( {\sigma_{3} - \sigma_{2}} \right)}{\sigma_{f}^{2}}{{\Delta\sigma}\left( {x,y} \right)}}}}\end{matrix} & (12)\end{matrix}$

The relationship shown in the formula (13) exists between the Defocusinformation B(x, y) and a difference Δσ(x, y) of the amount of blurringbetween the periphery and the center on the image plane. F in theformula (13) is the f-number.

$\begin{matrix}{{\Delta \; {\sigma \left( {x,y} \right)}} = \frac{B\left( {x,y} \right)}{F}} & (13)\end{matrix}$

The reason why the relationship of the formula (13) is established willbe described. The relation of f/D=B/Δσ exists from similitude relationsbetween figures formed of beams of light, the position of focus and thelike. f is the focal length and D is an effective diameter of the lens.Rearranging the relation yields Δσ=DB/f. Because the f-number is givenby F=f/D, the formula (13) is derived.

If the formula (13) is substituted into the formula (12) and rearranged,the formula (14) is derived.

$\begin{matrix}{{k_{comp}\left( {x,y} \right)} = {{k^{\prime}\left( {x,y} \right)} - \underset{}{\frac{2\left( {\sigma_{3} - \sigma_{2}} \right)}{\sigma_{f}^{2}}\frac{B\left( {x,y} \right)}{F}}}} & (14)\end{matrix}$

Correction Value

In the formula (14), the function and variables appearing on the rightside are all known. That is, the correction value necessary forprocessing can be obtained from known information. The correction valueis subtracted from the first distance information k′(x, y) to calculatethe distance information k_(—comp)(x, y), which is the second distanceinformation. The correction processing is performed on all pixels toobtain the distance information k_(—comp)(x, y) for each pixel. Then,the corrected distance map k_(—comp)(x, y) without error constituted ofthe distance information k_(—comp)(x, y) of all pixels is obtained. Inthe processing in step S8 of FIG. 4, the blurring processing byreferring to the corrected distance map k_(—comp)(x, y) is performed bythe blurring processing application unit 6.

2. Second Embodiment Distance Map Correction Processing in the SecondEmbodiment

Next, the second embodiment will be described. In the second embodiment,content of the distance map correction processing in step S7 of FIG. 4is different from content of the distance map correction processing inthe first embodiment. The configuration of the signal processingapparatus 2 and the processing in steps S6 and S8 of FIG. 4 and the likeare the same as in the first embodiment and thus, a duplicatedescription is omitted.

While the correction value is calculated by using the known Defocusinformation in the first embodiment, in the second embodiment, thecorrection value is estimated by using a distance map constituted offirst distance information. FIG. 23 is a flow chart exemplifying theflow of the distance map correction processing in the second embodiment.The distance map k′(x, y) constituted of the distance information k′(x,y) for all pixels is already generated by the processing in step S6 ofFIG. 4.

In step S22, foreground replacement processing is performed. Theforeground replacement processing replaces distance information ofpositions corresponding to a foreground subject in the distance mapk′(x, y) by using a portion of distance information of positionscorresponding to a background subject. For example, the distanceinformation is replaced by interpolation. A distance map k′_bg(x, y) isgenerated by the foreground replacement processing. Then, the processingproceeds to step S23.

In step S23, coefficients for a plurality of two-dimensional basesmodeling errors caused by, for example, the curvature of field andsingle-sided blurring for the distance map k′_bg(x, y) are calculated.If processing is performed in units of pixels, coefficients for aplurality of two-dimensional bases modeling errors of distanceinformation for each pixel in the distance map k′_bg(x, y) arecalculated. Then, the processing proceeds to step S24.

In step S24, a correction map is calculated as a linear sum usingcoefficients calculated in step S23 and bases. Then, the processingproceeds to step S25. In step S25, the correction map determined in stepS24 is subtracted from the distance map k′(x, y) on which errors aresuperimposed to correct errors. Then, the corrected distance mapk_(—comp)(x, y), which is a distance map after the correction, isgenerated.

The processing in each step of FIG. 23 will be described in detail. Theforeground replacement processing in step S22 includes, for example,foreground discrimination processing to discriminate the range of aforeground subject and distance information replacement processing toreplace distance information of positions corresponding to the range ofthe discriminated foreground subject with a portion of distanceinformation of positions corresponding to a background subject.

FIG. 24A exemplifies the distance map k′(x, y) before the correction. Inthe distance map k′(x, y), the rectangular location near the centerwhere the concentration is high is a foreground subject. The foregrounddiscrimination processing discriminates the range of a foregroundsubject by, for example, setting a threshold for the distanceinformation k′(x, y) for each pixel and comparing with the threshold.Instead of the threshold, the range of a foreground subject may bediscriminated by other methods. For example, the range of a foregroundsubject may be discriminated by applying a publicly known technique todiscriminate the range of persons in an image. Alternatively, an edgemay be detected by image processing to discriminate the range formed bythe detected edge as the range of a foreground subject.

After the range of a foreground subject being discriminated, thedistance information replacement processing is performed. As shown inFIG. 24B, distance information of positions corresponding to the rangeof the discriminated foreground subject is replaced with a portion ofdistance information of positions corresponding to a background subject.The distance information in the range of the foreground subject isreplaced with, for example, distance information of the periphery(neighborhood) of the range of the foreground subject by interpolation.The distance information used for distance information replacementprocessing is distance information obtained by appropriate distanceestimation. That is, in FIG. 24B, distance information estimated to beto the far side is used for the periphery of the range of the foregroundsubject. The distance map k′_bg(x, y) is generated by the distanceinformation replacement processing.

When coefficients corresponding to each base are calculated byprocessing in step S23 described later, it is difficult to appropriatelycalculate coefficients if the foreground and the background are mixed inthe whole distance map. Thus, the whole distance map is preferablyconfigured only by the background. Therefore, the processing in step S22is preferably performed.

Next, an error base used for processing in steps S23 and S24 will bedescribed. The shape of a curvature of field changes significantlydepending on the lens design and lens state (the focal length, f-number,and position of focus). FIG. 25 exemplifies the appearance (section)thereof. To estimate the amount of error of a curvature of fieldchanging in various ways superimposed on distance information in adistance map from the distance map, one or a plurality of maincomponents is extracted from various shapes of the curvature of field.One or the plurality of main components extracted may be stored inadvance.

Then, the amount of error contribution caused by each main component isestimated by calculating coefficients when these main components areused as bases from the distance map. FIGS. 26 and 27 show examples of abase P_1(x, y) by a first main component extracted and a base P_2(x, y)by a second main component extracted respectively. The bases shown inFIGS. 26 and 27 correspond to each of the shapes of the curvature offield shown in FIG. 25.

Though single-sided blurring in various directions can arise, if atleast two bases of single-sided blurring Q_1(x, y) in the verticaldirection and single-sided blurring Q_2(x, y) in the horizontaldirection are defined, single-sided blurring in various directions cansufficiently be represented as a linear sum of these bases. FIGS. 28 and29 show examples of respective bases. Bases illustrated in FIGS. 26 to29 may be stored in advance or supplied from outside the imagingapparatus 3 or the like. Information about bases stored or supplied fromoutside is acquired by the distance information correction unit 5 c.

In step S23, as shown in the following formulas (15) to (18),coefficients pc_1, pc_2, qc_1, qc_2 contained in the two-dimensionaldistance map k′_bg(x, y) obtained by replacing foreground distanceinformation with background distance information and corresponding torespective bases are calculated.

$\begin{matrix}{{pc}_{1} = {\sum\limits_{x}\; {\sum\limits_{y}\; {{P_{1}\left( {x,y} \right)}*{k_{bg}^{\prime}\left( {x,y} \right)}}}}} & (15) \\{{pc}_{2} = {\sum\limits_{x}\; {\sum\limits_{y}\; {{P_{2}\left( {x,y} \right)}*{k_{bg}^{\prime}\left( {x,y} \right)}}}}} & (16) \\{{qc}_{1} = {\sum\limits_{x}\; {\sum\limits_{y}\; {{Q_{1}\left( {x,y} \right)}*{k_{bg}^{\prime}\left( {x,y} \right)}}}}} & (17) \\{{qc}_{2} = {\sum\limits_{x}\; {\sum\limits_{y}\; {{Q_{2}\left( {x,y} \right)}*{k_{bg}^{\prime}\left( {x,y} \right)}}}}} & (18)\end{matrix}$

In step S24, a linear sum of each base as shown in the formula (19) isrepresented by using the coefficients determined in step S23. Theprocessing is performed for all pixels to generate a correction mapErr(x, y) for errors caused by the curvature of field and single-sidedblurring.

Err(x,y)=pc ₁ P ₁(x,y)+pc ₂ P ₂(x,y)+qc ₁ Q ₁(x,y)+qc₂ Q ₂(x,y)  (19)

Lastly, in step S25, as shown in the formula (20), the correction mapErr(x, y) is subtracted from the distance map k′(x, y). For example,distance information in the same position in the distance map k′(x, y)and the correction map Err(x, y) is subtracted. The processing shown inthe formula (20) is performed on distance information of all pixels togenerate the corrected distance map k_(—comp)(x, y) in which errorscaused by the curvature of field and single-sided blurring arecorrected.

$\begin{matrix}{{k_{comp}\left( {x,y} \right)} = {{k^{\prime}\left( {x,y} \right)} - \underset{}{{Err}\left( {x,y} \right)}}} & (20)\end{matrix}$

Correction Value 3. Third Embodiment

The present disclosure can also be configured as an imaging apparatus.FIG. 30 exemplifies the configuration of an imaging apparatus applicableto the present disclosure. The imaging apparatus includes a camera unit10, a digital signal processing unit 20, an SDRAM (Synchronous DynamicRandom Access Memory) 30, a medium interface (hereinafter, denoted as amedium I/F) 40, an operation unit 60, an LCD (Liquid Crystal Display)controller 70, an LCD 71, an external interface (hereinafter, denoted asan external I/F) 80, and a hard disk drive 90 as a large-capacityrecording medium and a recording medium 41 can be inserted into orremoved from the medium I/F 40.

The recording medium 41 is, for example, a so-called memory card using asemiconductor memory. In addition to the memory card, a hard diskapparatus, an optical recording medium such as a recordable DVD (DigitalVersatile Disc) and recordable CD (Compact Disc), or a magnetic disk canbe used. In the imaging apparatus, further a CPU 51 as an example of thecontrol unit, a RAM 52, a flash ROM 53, and a clock circuit 54 areconnected to a system bus 55.

The camera unit 10 includes an optical block 11, an image sensor 12 suchas a CCD or CMOS, a preprocessing circuit 13, an optical block driver14, an image sensor driver 15, and a timing generator 16. The opticalblock 11 includes lenses, a focusing mechanism, a shutter mechanism, andan iris mechanism.

The CPU 51 is, for example, a microcomputer and controls each unit ofthe imaging apparatus. The RAM 52 is mainly used as a work area such astemporarily storing intermediate results of processing. The flash ROM 53stores various programs executed by the CPU 51 and data necessary forprocessing. The Defocus information B(x, y), bases of main components,f-number of the imaging apparatus and the like are stored in the flashROM 53. The clock circuit 54 has a function to provide the current date(year/month/day), current day of week, current time, shooting date andtime and the like and also to add date/time information such as theshooting date and time to a shooting image file.

The optical block driver 14 forms a driving signal to operate theoptical block 11 in accordance with the control from the CPU 51 duringphotographing to operate the optical block 11 by supplying the drivingsignal to the optical block 11. In the optical block 11, the focusingmechanism, shutter mechanism, and iris mechanism are controlled inaccordance with the driving signal from the optical block driver 14 tocapture a subject image and then, the subject image is provided to theimage sensor 12.

The image sensor 12 performs a photoelectric conversion of the subjectimage from the optical block 11 and outputs the converted image. Theimage sensor 12 operates in accordance with the driving signal from theimage sensor driver 15 to capture the subject image and the capturedsubject image is supplied to the preprocessing circuit 13 as an electricsignal based on a timing signal from the timing generator 16 controlledby the CPU 51.

The timing generator 16 forms a timing signal that providespredetermined timing in accordance with the control from the CPU 51. Theimage sensor driver 15 forms a driving signal supplied to the imagesensor 12 based on a timing signal from the timing generator 16.

The preprocessing circuit 13 performs CDS (Correlated Double Sampling)processing on the supplied imaging signal to improve the S/N ratio,performs AGC (Automatic Gain Control) processing thereon to control thegain, and then performs an A/D (Analog/Digital) conversion to formimaging data as a digital signal.

The digital imaging data from the preprocessing circuit 13 is suppliedto the digital signal processing unit 20. The digital signal processingunit 20 performs camera signal processing such as de-mosaic processing,AF (Auto Focus), AE (Auto Exposure), and AWB (Auto White Balance) on theimaging data. The image data on which the camera signal processing hasbeen performed is compressed by a predetermined compression method andsupplied to the inserted recording medium 41 and/or the hard disk drive90 through the system bus 55 and the medium I/F 40 to record the imagedata in the recording medium 41 and/or the hard disk drive 90 as animage file conforming to, for example, the DCF (Design rule for CameraFile system) standard.

Intended image data of the image data recorded in the recording medium41 is read from the recording medium 41 through the medium I/F 40 inaccordance with operation input from a user accepted through theoperation unit 60 and the read image data is supplied to the digitalsignal processing unit 20. The operation unit 60 includes variousbuttons such as a shutter release button, levers, and dials. The LCD 71may be configured as a touch panel so that the user can perform an inputoperation by pressing on the screen using a finger or pointing device.

The digital signal processing unit 20 performs decompression processingof compression on the compressed image data read from the recordingmedium 41 and supplied through the medium I/F 40 to supply thedecompressed image data to the LCD controller 70 through the system bus55. The LCD controller 70 forms a display image signal supplied to theLCD 71 from the image data and supplies the display image signal to theLCD 71. Accordingly, an image in accordance with the image data recordedin the recording medium 41 is displayed in the screen of the LCD 71.Further, text such as a menu or graphics can be displayed in the screenof the LCD 71 under the control of the CPU 51 and the LCD controller 70.The form of the display in the screen follows a display processingprogram recorded in the flash ROM 53.

The imaging apparatus is provided with the external I/F 80. For example,an external personal computer is connected through the external I/F 80and image data can be supplied from the personal computer to record theimage data in the recording medium 41 inserted into the imagingapparatus or image data recorded in the recording medium 41 insertedinto the imaging apparatus can be supplied to the external personalcomputer.

For example, a network such as the Internet is connected by connecting acommunication module to the external I/F 80 and various kinds of imagedata and other information can be acquired through the network to recordsuch data in a recording medium inserted into the imaging apparatus ordata recorded in a recording medium inserted into the imaging apparatuscan be transmitted to an intended destination through the network.

Moreover, information such as image data acquired from an externalpersonal computer or through a network and recorded in a recordingmedium can be read and reproduced and displayed in the LCD 71.

Incidentally, the external I/F 80 can be provided as a wired interfacesuch as IEEE (Institute of Electrical and Electronics Engineers) 1394and USB (Universal Serial Bus) or as a wireless interface by light or byradio waves. That is, the external I/F 80 may be wired or wirelessinterface. For example, an external computer apparatus (not shown) isconnected through the external I/F 80 and image data supplied from thecomputer apparatus can be recorded in the recording medium 41 and/or thehard disk drive 90. Image data recorded in the recording medium 41and/or the hard disk drive 90 can also be supplied to an externalcomputer apparatus.

Subject images (still images and dynamic images) can be captured by theabove imaging apparatus to record the images in the inserted recordingmedium 41 and/or the hard disk drive 90. Further, image data recorded inthe recording medium 41 and/or the hard disk drive 90 can be read todisplay images or optionally browse or edit images. An index file tomanage image data is recorded in a specific region of the recordingmedium 41 and/or the hard disk drive 90.

The operation of the above imaging apparatus will briefly be described.A signal received by the image sensor 12 and photoelectrically convertedis supplied to the preprocessing circuit 13 in which the signalundergoes CDS processing and AGC processing and is converted into adigital signal before being supplied to the digital signal processingunit 20. Image quality correction processing is performed by the digitalsignal processing unit 20 on image data, which is temporarily stored inthe RAM 52 as image data of a camera-through image.

The image stored in the RAM 52 is supplied to the LCD controller 70under the control of the CPU 51 and a camera through image is displayedin the LCD 71. The angle of view can be adjusted while viewing thecamera through image displayed in the LCD 71. The image data maydirectly be supplied to the LCD controller 70 from the digital signalprocessing unit 20 without being stored in the RAM 52.

Then, when the shutter release button of the operation unit 60 ispressed, the CPU 51 outputs a control signal to the camera unit 10 tooperate the shutter of the optical block 11. At the same time, imagedata (recorded image data) for one frame supplied from the preprocessingcircuit 13 is processed by the digital signal processing unit 20 andstored in the SDRAM 30. Further, the recorded image data is compressedand encoded by the digital signal processing unit 20 and the encodeddata is stored in the hard disk 90 and also stored in the recordingmedium 41 through the system bus 55 and the medium I/F 4.

The CPU 51 may acquire the date/time or the time of photographing fromthe clock circuit 54 to add the acquired time information to still imagedata. Further, reduced image data of still images may further begenerated for still image data so that the still image data is stored inthe hard disk drive 90 and the recording medium 41 by being associatedwith the original still image data.

On the other hand, when recorded image data stored in the hard diskdrive 90 or the recording medium 41 is reproduced, recorded image dataselected by the CPU 51 is read from the SDRAM 30 in accordance withoperation input from the operation unit 60. The read recorded image datais decoded by the digital signal processing unit 20. The decoded imagedata is supplied to the LCD 71 through the LCD controller 70 and thereproduced image is displayed in the LCD 71.

FIG. 31 shows an example of the configuration of the CPU 51 of the aboveimaging apparatus and the like. The CPU 51 includes at least a distanceinformation calculation unit 56 and a distance information correctionunit 57. As shown in FIG. 31, the CPU 51 may be configured to include ablurring processing application unit 58. The CPU 51 may also beconfigured to include the RAM 52. The focus is controlled by a lens 11 ainside the optical block 11 being controlled by a focus controller 14 a.The focus controller 14 a includes the CPU 51 and the optical blockdriver 14.

A plurality of image shots of mutually different focusing is acquiredunder the control of the focus controller 14 a in the imaging apparatus.For example, an image (called a first image) in which a subject, forexample, a person in the foreground is in focus is acquired. Next, thesame subject is captured by shifting the focus to the far side by apredetermined distance with respect to the first image to acquire animage (called a second image). Further, the same subject is captured byshifting the focus to the near side by the predetermined distance withrespect to the first image to acquire an image (called a third image).These three images are temporarily stored in the RAM 52 as a datastorage unit.

The second image and the third image of the three images are used by thedistance information calculation unit 56 to calculate, for example,distance information for each pixel. Then, a distance map constituted ofdistance information of all pixels is generated. The processing by thedistance information calculation unit 56 is the same as the processingby the distance information calculation unit 5 b described above andthus, a detailed description thereof is omitted.

The distance map generated by the distance information calculation unit56 is supplied to the distance information correction unit 57. Thedistance information correction unit 57 corrects the distanceinformation for each pixel in the supplied distance map. Then, acorrected distance map constituted of distance information of all pixelsafter the correction is generated. The processing by the distanceinformation correction unit 57 is the same as the processing by thedistance information correction unit 5 c in the first or secondembodiment described above and thus, a detailed description thereof isomitted. The respective processing of the distance informationcorrection unit 5 c in the first and second embodiments may be set asmodes so that the mode is switched and performed by the distanceinformation correction unit 57.

The corrected distance map generated by the distance informationcorrection unit 57 is supplied to the blurring processing applicationunit 58. The blurring processing application unit 58 performs blurringprocessing referring to the corrected distance map and, for example, animage suitable for appreciation with a blurred background is generated.The processing by the blurring processing application unit 58 is thesame as the processing by the blurring processing application unit 6 andthus, a detailed description thereof is omitted. The generated image maybe supplied to the LCD 71 via the system bus 55 and the LCD controller70. As described above, the present disclosure can be configured as, forexample, an imaging apparatus.

4. Modification

In the foregoing, a plurality of embodiments of the present disclosurehas been concretely described, but the present disclosure is not limitedto these embodiments. In the above embodiments, for example, applicationexamples to errors caused by the curvature or field or single-sidedblurring are shown, but the present disclosure may similarly be appliedto other kinds of aberration.

Further, in the above embodiments, a plurality of images is acquired bychanging the position of focus and then, a distance map is generatedbased on differences of blurring functions of these images. However, aplurality of images may be acquired by changing the iris to generate adistance map based on differences of blurring functions of these images.Further, in the above embodiments, examples of using a distance map forblurring processing are shown. In addition, processing to generate astereo image from a plurality of parallax images or generationprocessing of a field depth extended image by deconvolution inaccordance with the distance may also be performed by using the distancemap. Also, degradation in quality of an appreciation image caused by thecurvature of field or single-sided blurring may be corrected bydeconvolution by using the distance map.

In the above embodiments, content of processing in the second embodimentmay adaptively be changed in accordance with the distribution of, forexample, distance information equal to a threshold or more in thedistance map obtained in step S6 of FIG. 4. It is assumed, for example,that the distance map shown in FIG. 16 or FIG. 18 is obtained by stepS6. Because an error in distance estimation is caused by the curvatureof field in this case, the coefficients pc_1, pc_2 may be calculated,instead of coefficients for all bases. If the distance map shown in FIG.20 is obtained by step S6, the coefficients qc_1, qc_2 may be calculatedbecause an error in distance estimation is caused by single-sidedblurring. Accordingly, the amount of computation in the distance mapcorrection processing can be reduced.

In the above embodiments, the control unit 5 in the signal processingapparatus 2 may be configured by a distance information input unit intowhich first distance information is input and the distance informationcorrection unit 5 c that corrects the input distance information.

Configurations and content of processing in the plurality of embodimentsand the modification described above may mutually applied as long astechnical inconsistency does not arise. Further, the present disclosurecan be configured, in addition to the apparatus, as a method, a program,or a recording medium recording the program.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

The present technology may also be configured as below.

(1) A signal processing apparatus, comprising:

an input unit into which first image data and second image data areinput, the first image data and the second image data being obtained bycapturing a predetermined subject with an imaging unit and havingmutually different states of blurring;

a distance information calculation unit that calculates first distanceinformation in accordance with a position in a depth direction of thesubject based on the first image data and the second image data; and

a distance information correction unit that calculates second distanceinformation by performing, on the first distance information, processingto correct a deviation of the blurring caused by a mismatch between animage-forming plane of the subject and an imaging plane of the imagingunit.

(2) The signal processing apparatus according to (1),

wherein the distance information correction unit calculates a correctionvalue to correct a deviation of the blurring by using known informationand calculates the second distance information by performing, on thefirst distance information, processing using the correction value.

(3) The signal processing apparatus according to (1),

wherein the distance information correction unit calculates a correctionvalue to correct a deviation of the blurring by using the first distanceinformation and calculates the second distance information byperforming, on the first distance information, processing using thecorrection value.

(4) The signal processing apparatus according to (3),

wherein the distance information correction unit acquires a plurality ofbases modeling an error superimposed on the first distance informationand calculates the correction value to correct the deviation of theblurring by using the first distance information and the plurality ofbases.

(5) The signal processing apparatus according to (4),

wherein the distance information correction unit calculates thecorrection value to correct the deviation of the blurring byrepresenting the first distance information as a linear sum of theplurality of bases.

(6) The signal processing apparatus according to (3),

wherein the subject includes a foreground subject and a backgroundsubject, and

the distance information correction unit replaces the first distanceinformation in positions corresponding to the foreground subject byusing a portion of the first distance information in positionscorresponding to the background subject and calculates the correctionvalue to correct the deviation of the blurring by using the firstdistance information in positions corresponding to the backgroundsubject and the first distance information after being replaced.

(7) The signal processing apparatus according to any one of (1) to (6),

wherein the first image data is obtained by capturing the subject in afirst optical unit state of the imaging unit,

the second image data is obtained by capturing the subject in a secondoptical unit state of the imaging unit, and

the first optical unit state and the second optical unit state aremutually different states of a position of focus.

(8) The signal processing apparatus according to any one of (1) to (6),

wherein the first image data is obtained by capturing the subject in afirst optical unit state of the imaging unit,

the second image data is obtained by capturing the subject in a secondoptical unit state of the imaging unit, and

the first optical unit state and the second optical unit state aremutually different states of an iris.

(9) The signal processing apparatus according to any one of (1) to (8),

wherein the distance information calculation unit performs blurringprocessing of adding the blurring to the first image data through afilter, determines a similarity between the first image data to whichthe blurring has been added and the second image data, detects thenumber of times of the blurring processing when the similarity ismaximum, and calculates the first distance information from the detectednumber of times of the blurring processing.

(10) The signal processing apparatus according to (9), wherein thenumber of times of the blurring processing that adds no blurring is setas 0.(11) An imaging apparatus, comprising:

an imaging unit;

an input unit into which first image data and second image data areinput, the first image data and the second image data being obtained bycapturing a predetermined subject with the imaging unit and havingmutually different states of blurring;

a distance information calculation unit that calculates first distanceinformation in accordance with a position in a depth direction of thesubject based on the first image data and the second image data; and

a distance information correction unit that calculates second distanceinformation by performing, on the first distance information, processingto correct a deviation of the blurring caused by a mismatch between animage-forming plane of the subject and an imaging plane of the imagingunit.

(12) A signal processing method for a signal processing apparatus, themethod comprising:

inputting first image data and second image data, the first image dataand the second image data being obtained by capturing a predeterminedsubject with an imaging unit and having mutually different states ofblurring;

calculating first distance information in accordance with a position ina depth direction of the subject based on the first image data and thesecond image data; and

calculating second distance information by performing, on the firstdistance information, processing to correct a deviation of the blurringcaused by a mismatch between an image-forming plane of the subject andan imaging plane of the imaging unit.

(13) A program causing a computer to execute a signal processing methodfor a signal processing apparatus, the method comprising:

inputting first image data and second image data, the first image dataand the second image data being obtained by capturing a predeterminedsubject with an imaging unit and having mutually different states ofblurring;

calculating first distance information in accordance with a position ina depth direction of the subject based on the first image data and thesecond image data; and

calculating second distance information by performing, on the firstdistance information, processing to correct a deviation of the blurringcaused by a mismatch between an image-forming plane of the subject andan imaging plane of the imaging unit.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2011-164266 filed in theJapan Patent Office on Jul. 27, 2011, the entire content of which ishereby incorporated by reference.

What is claimed is:
 1. A signal processing apparatus, comprising: aninput unit into which first image data and second image data are input,the first image data and the second image data being obtained bycapturing a predetermined subject with an imaging unit and havingmutually different states of blurring; a distance informationcalculation unit that calculates first distance information inaccordance with a position in a depth direction of the subject based onthe first image data and the second image data; and a distanceinformation correction unit that calculates second distance informationby performing, on the first distance information, processing to correcta deviation of the blurring caused by a mismatch between animage-forming plane of the subject and an imaging plane of the imagingunit.
 2. The signal processing apparatus according to claim 1, whereinthe distance information correction unit calculates a correction valueto correct a deviation of the blurring by using known information andcalculates the second distance information by performing, on the firstdistance information, processing using the correction value.
 3. Thesignal processing apparatus according to claim 1, wherein the distanceinformation correction unit calculates a correction value to correct adeviation of the blurring by using the first distance information andcalculates the second distance information by performing, on the firstdistance information, processing using the correction value.
 4. Thesignal processing apparatus according to claim 3, wherein the distanceinformation correction unit acquires a plurality of bases modeling anerror superimposed on the first distance information and calculates thecorrection value to correct the deviation of the blurring by using thefirst distance information and the plurality of bases.
 5. The signalprocessing apparatus according to claim 4, wherein the distanceinformation correction unit calculates the correction value to correctthe deviation of the blurring by representing the first distanceinformation as a linear sum of the plurality of bases.
 6. The signalprocessing apparatus according to claim 3, wherein the subject includesa foreground subject and a background subject, and the distanceinformation correction unit replaces the first distance information inpositions corresponding to the foreground subject by using a portion ofthe first distance information in positions corresponding to thebackground subject and calculates the correction value to correct thedeviation of the blurring by using the first distance information inpositions corresponding to the background subject and the first distanceinformation after being replaced.
 7. The signal processing apparatusaccording to claim 1, wherein the first image data is obtained bycapturing the subject in a first optical unit state of the imaging unit,the second image data is obtained by capturing the subject in a secondoptical unit state of the imaging unit, and the first optical unit stateand the second optical unit state are mutually different states of aposition of focus.
 8. The signal processing apparatus according to claim1, wherein the first image data is obtained by capturing the subject ina first optical unit state of the imaging unit, the second image data isobtained by capturing the subject in a second optical unit state of theimaging unit, and the first optical unit state and the second opticalunit state are mutually different states of an iris.
 9. The signalprocessing apparatus according to claim 1, wherein the distanceinformation calculation unit performs blurring processing of adding theblurring to the first image data through a filter, determines asimilarity between the first image data to which the blurring has beenadded and the second image data, detects the number of times of theblurring processing when the similarity is maximum, and calculates thefirst distance information from the detected number of times of theblurring processing.
 10. The signal processing apparatus according toclaim 9, wherein the number of times of the blurring processing thatadds no blurring is set as
 0. 11. An imaging apparatus, comprising: animaging unit; an input unit into which first image data and second imagedata are input, the first image data and the second image data beingobtained by capturing a predetermined subject with the imaging unit andhaving mutually different states of blurring; a distance informationcalculation unit that calculates first distance information inaccordance with a position in a depth direction of the subject based onthe first image data and the second image data; and a distanceinformation correction unit that calculates second distance informationby performing, on the first distance information, processing to correcta deviation of the blurring caused by a mismatch between animage-forming plane of the subject and an imaging plane of the imagingunit.
 12. A signal processing method for a signal processing apparatus,the method comprising: inputting first image data and second image data,the first image data and the second image data being obtained bycapturing a predetermined subject with an imaging unit and havingmutually different states of blurring; calculating first distanceinformation in accordance with a position in a depth direction of thesubject based on the first image data and the second image data; andcalculating second distance information by performing, on the firstdistance information, processing to correct a deviation of the blurringcaused by a mismatch between an image-forming plane of the subject andan imaging plane of the imaging unit.
 13. A program causing a computerto execute a signal processing method for a signal processing apparatus,the method comprising: inputting first image data and second image data,the first image data and the second image data being obtained bycapturing a predetermined subject with an imaging unit and havingmutually different states of blurring; calculating first distanceinformation in accordance with a position in a depth direction of thesubject based on the first image data and the second image data; andcalculating second distance information by performing, on the firstdistance information, processing to correct a deviation of the blurringcaused by a mismatch between an image-forming plane of the subject andan imaging plane of the imaging unit.